Stacked on pr3-harness-bench (#46). One file: the BrowseComp+ baseline agent.

The scoring bug

When a completion came back with finish_reason == "length", the agent submitted message.content verbatim as its final answer:

if not calls:
    if (message.content or "").strip():
        await self._submit(environment, message.content)

This target writes its chain-of-thought into ordinary content (it has no separate reasoning_content channel) and routinely runs past the token cap. So what got submitted was not an answer, it was 42k to 52k characters of truncated, mid-sentence reasoning. The judge scores that 0 every time, and the trial is indistinguishable from a genuine wrong answer.

Measured across the smoke runs carrying this change: 6 length-truncated turns over 3 trials, content length min 42,253 / median 46,793 / max 52,104 chars. The pre-fix baseline traces (198 trials over 3 rounds) contain no finish_reason field at all, which is exactly why this went unnoticed. The instrumentation added here is what surfaced it.

Truncated turns are now recoverable rather than final: the model is asked for only the three required labeled lines instead of having its unfinished thinking submitted on its behalf.

Supporting changes

All four target the same failure mode of losing a trial that had real retrieved evidence behind it.

Wall-clock deadline with a final-answer reserve. Turn count is not this agent's binding constraint. Most of a turn is sequential search.py invocations, each booting a fresh pyserini/Lucene JVM against the BM25 index (~13s apiece) against ~3-9s of inference. Trials ran out of clock long before MAX_TURNS, so the existing turn-budget fallback never fired and harbor killed the trial mid-search with no answer recorded. There is now a deadline (BROWSECOMP_AGENT_BUDGET_SEC, default 900) with a reserve (BROWSECOMP_FINAL_RESERVE_SEC, default 120) so the best-effort answer is written before harbor's AgentTimeoutError. Harbor does not tell an agent its own timeout, hence the env var; keep it in step with the run's --agent-timeout-multiplier.

Tool calls past the deadline are stubbed, never skipped, because every tool_call still needs a matching tool message or the forced-final request is rejected for an unanswered call.

Completion shape validation. A gateway can answer 200 with a body the SDK does not turn into a completion, observed once as 'str' object has no attribute 'usage', which crashed the trial on an attribute access two lines later. _complete() now rejects anything without usable .choices in one place.

Empty turns nudge instead of raising. raise RuntimeError("model returned neither a response nor a tool call") became a bounded retry, giving up only after MAX_EMPTY_RETRIES consecutive empty turns. An assistant message carrying neither content nor tool calls is dropped rather than appended, since strict OpenAI-compatible servers reject it.

Bounded forced-final. The last-resort call uses max_retries=1 (the SDK default of 8 could outlast the reserve on its own) and ANSWER_MAX_TOKENS (so the model cannot spend the whole reserve on chain-of-thought and get truncated a second time), and degrades to a stub answer on failure instead of propagating.

Extra trace fields. finish_reason, per-turn llm_sec, left_sec, and a forced_final_reason of deadline vs turns, which lets a post-hoc conservative score treat exactly the trials the pre-fix agent would have lost as zeros.

Rebased onto the current base

pr3-harness-bench moved (and its history was rewritten) after this branch was cut, so the original commit no longer applied. Rebuilt as one commit on 5727a44.

The _is_reasoning_model overlap flagged in the first version of this PR is resolved: #54's version of that gate has landed upstream and is kept verbatim here. All this PR now does to _completion_kwargs is route the already-computed token-limit key through a new max_tokens parameter, which the bounded forced-final call needs:

-        kwargs[_token_limit_key] = 12_000
+        kwargs[_token_limit_key] = max_tokens

Also fixed in the rebase

The first version broke the existing test suite. _complete reaches the client through with_options(...), and the test fake was a bare SimpleNamespace, so test_agent_submits_response_and_populates_context died on AttributeError: 'types.SimpleNamespace' object has no attribute 'with_options'. The fake is now a small FakeClient that implements it. Nothing in CI runs these target test suites (the repo has no workflows; the only checks are Greptile and Socket), which is why a red suite was not visible on the PR.

Three tests added, each verified to fail without its fix:

Both Greptile findings addressed:

1. Stubbed tool calls produced no trace events. The per-call _trace sat inside the executed branch, so deadline-stubbed calls were invisible in the JSONL exactly when a post-hoc audit needs them. Moved out so both paths are traced.
2. forced_final_reason conflated two causes. empty_turns >= MAX_EMPTY_RETRIES also reported "turns", so a trial that gave up on turn 4 after three truncated turns was indistinguishable from one that used all 32. There is now a third value, empty_retries, and the reason is computed once and used by both the trace and context.metadata.

Verification

• pytest tests/ in browsecomp-plus/baseline/target: 5 passed.
• pytest tests/test_v05_benchmark_configs.py: 13 passed, 6 skipped.
• Each new test confirmed red with its fix reverted and green with it restored.
• Exercised live in the smoke-browsecomp-plus-qwen fix1/fix2 runs cited above.

🤖 Generated with Claude Code

Greptile Summary

This PR fixes a scoring bug in the BrowseComp+ baseline agent where a finish_reason == "length" response (truncated mid-sentence reasoning) was submitted verbatim as the final answer, scoring 0 every time. It also adds a wall-clock deadline to prevent harbor from killing trials mid-search with no answer recorded.

• Headline fix: finish_reason == "length" turns are now treated as empty retries (with a nudge to produce only the three labeled lines) rather than submitted as the answer; after MAX_EMPTY_RETRIES consecutive truncated/empty turns the agent falls through to the forced-final path.
• Deadline guard: AGENT_BUDGET_SEC / FINAL_RESERVE_SEC env-var constants bound wall-clock time; tool calls past the deadline are stubbed (not skipped) so the forced-final request is never rejected for unanswered tool calls.
• Robustness hardening: _complete() validates response shape before attribute access, the forced-final call uses max_retries=1 and ANSWER_MAX_TOKENS to avoid re-truncation, and forced_final_reason now distinguishes deadline / empty_retries / turns for post-hoc scoring.
• Test suite: FakeClient.with_options repairs the pre-existing broken test; two new regression tests (TruncatedThenAnswer, AlwaysTruncated) were each verified red before their respective fixes.

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[run starts] --> B{remaining le FINAL_RESERVE_SEC?}
    B -->|yes| DEADLINE[out_of_time=True, break]
    B -->|no| C[_complete with timeout]
    C --> D{finish_reason}
    D -->|length truncated=True| E{empty_turns ge MAX_EMPTY_RETRIES?}
    D -->|stop with content| SC[_submit content, completed=True]
    D -->|tool_calls| T[process tool calls]
    E -->|yes| OOA[out_of_answers=True, break]
    E -->|no| NUDGE[append nudge message, continue]
    NUDGE --> B
    T --> DC{deadline hit mid-turn?}
    DC -->|yes stub result| TR[_trace event both paths]
    DC -->|no execute tool| TR
    TR --> SR{submit_response called?}
    SR -->|yes| DONE[completed=True, break]
    SR -->|no| OT{out_of_time?}
    OT -->|yes| DEADLINE
    OT -->|no| B
    DEADLINE --> FF
    OOA --> FF
    FF[forced-final: reason = deadline or empty_retries or turns]
    FF --> FFC[_complete tools=False, max_retries=1, ANSWER_MAX_TOKENS]
    FFC -->|success| FS[_submit answer]
    FFC -->|Exception| STUB[_submit stub answer]
    FS --> META[context.metadata forced_final_reason]
    STUB --> META

_{Reviews (3): Last reviewed commit: "fix: stop submitting truncated reasoning..." | Re-trigger Greptile}

Context used:

• Rule used - Store magic numbers as class or instance variables... (source)

Learned From
scaleapi/scaleapi#126388